LLaDA: Large Language Diffusion Models

LLaDA challenges the traditional autoregressive next-token prediction LLMs. It introduces LLaDA, a diffusion-based language model with masking. In short, tokens are progressively masked during training, and a bidirectional Transformer learns to predict all masked tokens simultaneously by optimizing a proper likelihood lower bound rather than a cross-entropy objective.